-
Notifications
You must be signed in to change notification settings - Fork 48
Support multiple actors on the same rank #494
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
samlurye
wants to merge
2
commits into
pytorch-labs:main
Choose a base branch
from
samlurye:export-D78047960
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
+1,179
−340
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Pull Request resolved: pytorch-labs#456 This diff contains several updates to the monarch actor mesh debugging experience. - Improved debug input parsing, with the new `cast` command supporting more sophisticated rank selection grammar: - `cast ranks(3) pdb_command`: send `pdb_command` to rank 3. - `cast ranks(1,3,5) pdb_command`: send `pdb_command` to ranks 1, 3 and 5. - `cast ranks(1:10:2) pdb_command`: send `pdb_command` to the ranks in `range(start=1, stop=10, step=2)`. - `cast ranks(pp=2, dp=(1,3), tp=2:8) pdb_command`: send `pdb_command` to ranks with `pp` dim 2, `dp` dim 1 or 3, and `tp` dim in `range(2,8)`. - The debug client is now automatically registered with an actor mesh when that actor mesh is spawned. This means calling `init_debugging(actor_mesh)` is no longer necessary. - Debugging now works with MAST jobs, by enforcing that breakpoints aren't set in `__main__`, and that the file containing the breakpoint exists on the remote host. - The first requirement is due to how `cloudpickle` works -- if an actor endpoint is defined inside `__main__`, `cloudpickle` will serialize it by value instead of by reference. When the code then runs on the remote host, it thinks the location of the code is the user's local `__main__` file, which confuses pdb, because the file doesn't exist at the same path (or may not exist at all) on the remote host. - The second requirement is due to important parts of `pdb`'s implementation relying on the ability to search for the file being debugged on the remote host's file system. - A debugging session for a specific rank is now forcefully exited once the endpoint finishes execution. This contains the debugging experience within user-authored code. It is also necessary for preventing hangs, because if pdb is allowed to continue indefinitely, then control flow will eventually bubble back up to the main asyncio event loop on the worker, at which point everything breaks. - Hitting a breakpoint now automatically enables post-mortem debugging, so any rank that encounters an exception after hitting a breakpoint will automatically stop at the exception. Attaching the debugger to that rank should then provide an experience like `pdb.post_mortem()`. ## Next steps/gaps I'm aware of (reviewers please read): - Indexing debug sessions by rank isn't sustainable, because two actor meshes may simultaneously hit breakpoints on the same rank and cause a collision inside the debug client. - Entering the debug client should happen automatically, rather than requiring the user to do `await debug_client().enter.call_one()`. - Casting pdb commands should ideally leverage `MeshTrait` rather than reimplementing the selection logic. - If a mesh was reshaped/renamed so that its dimension names aren't `hosts` and `gpus` anymore, the debugger should reflect the new shape/names. - The user should be able to enable post-mortem debugging without having to hit a separate breakpoint first. Differential Revision: D77568423 Reviewed By: zdevito
This pull request was exported from Phabricator. Differential Revision: D78047960 |
samlurye
pushed a commit
to samlurye/monarch-1
that referenced
this pull request
Jul 10, 2025
Summary: Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group. Differential Revision: D78047960
1ce0f62
to
59ba694
Compare
samlurye
pushed a commit
to samlurye/monarch-1
that referenced
this pull request
Jul 10, 2025
Summary: Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group. Differential Revision: D78047960
59ba694
to
cefbf99
Compare
This pull request was exported from Phabricator. Differential Revision: D78047960 |
samlurye
pushed a commit
to samlurye/monarch-1
that referenced
this pull request
Jul 10, 2025
Summary: Pull Request resolved: pytorch-labs#494 Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group. Differential Revision: D78047960
cefbf99
to
77be81a
Compare
Summary: Pull Request resolved: pytorch-labs#494 Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group. Differential Revision: D78047960
This pull request was exported from Phabricator. Differential Revision: D78047960 |
77be81a
to
79cf869
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The
attach
andcast
commands now require the user to specify actor name in addition to rank. Thelist
command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group.Differential Revision: D78047960